[feat] add preload the dir and config the depth of load.#1069
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
There was a problem hiding this comment.
Pull Request Overview
This PR introduces a new configuration option to limit directory preloading depth and implements related preloading functionality across the codebase, along with an API endpoint to retrieve a directory’s hash.
- Adds the "load_dir_depth" config key and helper function.
- Implements directory preloading in the async IO flow and updates directory hash accordingly.
- Introduces a new API route and corresponding service method for retrieving a directory’s hash.
Reviewed Changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| scorpio/src/util/config.rs | Adds "load_dir_depth" configuration key and helper function. |
| scorpio/src/dicfuse/tree_store.rs | Introduces a getter for parent and a method to update item hash. |
| scorpio/src/dicfuse/async_io.rs | Integrates directory loading depth calculation and updates hash logic. |
| scorpio/scorpio.toml | Adds "load_dir_depth" to the configuration file. |
| scorpio/Cargo.toml | Adds the dashmap dependency. |
| mono/src/api/api_router.rs | Adds a new API route for retrieving a directory’s hash. |
| ceres/src/api_service/mod.rs | Implements the corresponding API method to return directory hash data. |
|
Attempt to consolidate methods and structures related to external interfaces into the |
| dicfuse_readable = "true" No newline at end of file | ||
| base_url = "http://localhost:8000" | ||
| dicfuse_readable = "true" | ||
| load_dir_depth = "3" No newline at end of file |
There was a problem hiding this comment.
Improve the description of configuration options in the README of scorpio.
| mod abi; | ||
| mod async_io; | ||
| mod store; | ||
| pub mod store; |
There was a problem hiding this comment.
Is it necessary to expose this store module? I think only dicfuse should use it.
There was a problem hiding this comment.
Pull Request Overview
This PR introduces new functionality for preloading directories and configuring the depth of load, along with an API to return a directory’s hash.
- Introduces a new configuration key "load_dir_depth" with corresponding helper function.
- Exposes and updates the dicfuse module and its access methods to include directory hash updates.
- Updates the mono API endpoints and documentation to support the new preload and hash features.
Reviewed Changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| scorpio/src/util/config.rs | Adds "load_dir_depth" config and getter function. |
| scorpio/src/lib.rs | Makes the dicfuse module public. |
| scorpio/src/dicfuse/tree_store.rs | Comments out unused constant and adds update_item_hash function. |
| scorpio/src/dicfuse/mod.rs | Exposes the store module publicly. |
| scorpio/src/dicfuse/async_io.rs | Updates access function to preload directory hash and remove debug prints. |
| scorpio/scorpio.toml | Updates configuration with new keys and local URLs. |
| scorpio/README.md | Updates configuration guide with new options. |
| scorpio/Cargo.toml | Updates dependencies and dev-dependencies to support changes. |
| mono/src/api/api_router.rs | Adds a new endpoint for obtaining a directory’s hash. |
| ceres/src/api_service/mod.rs | Implements API handler for retrieving a directory’s hash. |
| Cargo.toml | Adds scorpio to the default workspace members. |
There was a problem hiding this comment.
Pull Request Overview
This PR introduces a configurable preload depth for directory loading, integrates on-demand preloading in the FUSE access hook, and adds a new API endpoint to fetch a directory’s hash.
- Add
load_dir_depthconfig entry with default, validation, and accessor. - Make
dicfuse’s store module public and inject preload logic intoFilesystem::access. - Expose a new
/tree/dir-hashroute and implementget_tree_dir_hashin both mono router and ceres handler.
Reviewed Changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| scorpio/src/util/config.rs | Added load_dir_depth default, validation, and load_dir_depth() |
| scorpio/src/lib.rs | Changed dicfuse from private to public module |
| scorpio/src/dicfuse/tree_store.rs | Removed old config-file logic and added update_item_hash |
| scorpio/src/dicfuse/mod.rs | Made store module public |
| scorpio/src/dicfuse/async_io.rs | Added on-access preload using load_dir and hash update logic |
| scorpio/scorpio.toml | Configured load_dir_depth, updated URLs |
| scorpio/README.md | Documented load_dir_depth and dicfuse_readable |
| scorpio/Cargo.toml | Added dashmap and various dev-dependencies |
| mono/src/api/api_router.rs | Registered get_tree_dir_hash route |
| ceres/src/api_service/mod.rs | Implemented get_tree_dir_hash on the ApiHandler trait |
| Cargo.toml | Added scorpio to workspace members |
Comments suppressed due to low confidence (3)
scorpio/src/util/config.rs:247
- The new
load_dir_depth()accessor isn’t covered by tests. Consider adding a unit test to verify parsing, missing values, and fallback logic.
pub fn load_dir_depth() -> usize {
scorpio/src/dicfuse/async_io.rs:141
- The new preload logic and hash-change flow in the
accessmethod lack tests. You should add integration or unit tests to verify hash updates under various directory structures.
let hash_change = load_dir(self.store.clone(), load_parent, max_depth).await;
ceres/src/api_service/mod.rs:220
- There are no tests for
get_tree_dir_hash. Adding unit tests for both found and empty-path scenarios will ensure correct behavior and prevent regressions.
async fn get_tree_dir_hash(
| config.insert("git_author".to_string(), "MEGA".to_string()); | ||
| config.insert("git_email".to_string(), "admin@mega.org".to_string()); | ||
| config.insert("config_file".to_string(), "config.toml".to_string()); | ||
| config.insert("load_dir_depth".to_string(), "3".to_string()); |
There was a problem hiding this comment.
The default depth 3 is hard-coded in multiple places. Extract this to a shared constant to avoid duplication and keep defaults in sync.
| config.insert("load_dir_depth".to_string(), "3".to_string()); | |
| config.insert("load_dir_depth".to_string(), DEFAULT_LOAD_DIR_DEPTH.to_string()); |
| db: Db, | ||
| } | ||
| const CONFIG_PATH: &str = "config.toml"; | ||
| // const CONFIG_PATH: &str = "config.toml"; |
There was a problem hiding this comment.
Commented out code (CONFIG_PATH and related imports) should be removed to keep the file clean and avoid confusion.
| // const CONFIG_PATH: &str = "config.toml"; |
| } | ||
|
|
||
| /// return the dir's hash only | ||
| async fn get_tree_dir_hash( |
There was a problem hiding this comment.
[nitpick] This method is quite long and handles multiple responsibilities (tree search, mapping, commit lookup). Consider splitting into helper functions to improve readability and testability.
|
LGTM @genedna |
1.首次从mono加载，只加载指定深度的目录
2.每隔一段时间，更新已经加载的目录
3.用户访问到一个路径，就预加载指定深度的子路径
4.mono添加一个api，直接返回给定路径文件夹的hash值